feat(bots): onboard reviewer-bot to pinned databricks-bot-engine#862
Conversation
|
🚧 Held as draft pending an engine-side fix. This onboarding is blocked on databricks/databricks-bot-engine#97: the reusable
The engineer-bot onboarding (bug-fix flow, PAT-free via its own job) will be Prerequisite for either bot: the review/engineer GitHub App must be installed on |
There was a problem hiding this comment.
Verdict: 1 High · 1 Medium · 2 Low
Mostly workflow/config plumbing for onboarding the review + engineer bots; no product (src/) code changes. Main concerns: a TEMP e2e workflow the PR itself marks DO-NOT-MERGE is still in the diff (F1), a stray doubly-nested setup-jfrog/setup-jfrog action that nothing references and duplicates the real one (F2), plus a couple of comment/outcome-string mismatches (F3, F4).
There was a problem hiding this comment.
Verdict: 1 Medium · 1 Low
Onboarding PR is almost entirely CI/workflow/config with no connector src/ changes, so risk is scoped to the GitHub Actions. Two issues worth addressing before merge: a self-declared TEMP "DO NOT MERGE" e2e workflow that is still in the diff (medium), and an orphaned double-nested setup-jfrog/setup-jfrog/action.yml duplicate that nothing references (low). Nit: the reviewer-bot.yml header comment still says the token is passed "as engine-pat to install-bot-engine", but the composite input is engine-token — stale wording, worth a one-line fix.
✅ Ready for review — both bots onboarded and verified liveSquashed to 3 signed-off commits (DCO green). Both bots run against the upstream Verified end-to-end:
Key design point (first clean consumer): an external repo cannot Superseded: the vendored reviewer-bot stack (#832/#833/#834) — closed. |
There was a problem hiding this comment.
Verdict: 1 Medium
Mostly workflow/config plumbing that reads carefully and is well-commented. One medium concern: the engineer-bot issue-comment case matches no_change, while the prompt and the design doc's live-verification note both use the token no_change_needed — a mismatch that would mislabel a real no-change outcome as a failure. Worth confirming the engine's exact outcome-enum string before merge.
There was a problem hiding this comment.
Verdict: 1 Medium · 1 Low
Solid infra onboarding PR — no security or correctness issues in the token-minting / gating logic, which is carefully done. One medium concern: the issue-comment case in engineer-bot.yml matches no_change but the author actually emits no_change_needed (per this PR's own prompt and design doc), so no-change runs post a misleading "could not complete a fix" comment. Plus one low: the coverage_pr_url output name looks like a copy-paste from a coverage flow and should be confirmed against the engine's bug-fix publish output.
There was a problem hiding this comment.
Verdict: 1 Low
Looks good overall — this is CI/workflow + .bot/ prompt plumbing with no src/ changes, so the correctness/test axes are limited. Vendored-code removal is clean (grep confirms no dangling scripts.reviewer_bot/scripts.shared references), token handling in install-bot-engine is careful (job-local git extraheader, masking, SHA pin validation), and the security gates are sound. One low-severity gate inconsistency noted inline.
Nits (no inline anchor needed): (1) install-bot-engine/action.yml hardcodes sdk-version: 0.2.102 / cli-version: 2.1.61 defaults that must be manually kept in sync with the engine at the pinned SHA — the comments flag this, but it's a standing drift risk worth a periodic check. (2) The JFrog OIDC block relies on the friendly ACCESS_TOKEN empty-check for error messaging, but under set -euo pipefail a missing ACTIONS_ID_TOKEN_REQUEST_TOKEN would abort at the curl | jq pipe before that check fires; since a
[...truncated to keep verdict scannable]
There was a problem hiding this comment.
Verdict: 2 Low
Looks good overall — a clean, well-documented bot-onboarding PR (workflow YAML, local composite actions, prompts, config; no product code touched). Two low-severity consistency notes: the bot-prelude engine-repo input isn't honored by the engine-token mint step, and the initial reviewer workflow lacks the per-PR concurrency group its siblings have. Security-sensitive paths (untrusted issue/PR input via files + env heredocs, digits-only validation, PAT-free git-auth masking, fork gates) are handled carefully.
There was a problem hiding this comment.
Verdict: 1 Low
Looks good — clean, well-documented onboarding PR. Vendored-bot deletions are confirmed (no dangling references), the four workflows are internally consistent (shared bot-prelude/install-bot-engine composites, PAT-free App-auth, consistent MODEL_ENDPOINT, bot-login prefixes matching .bot/config.yaml), and the untrusted-input handling (digits-only issue/PR validation, heredoc-delimiter collision guard, issue body via file not shell) is careful. One low-confidence note about the workflow_dispatch checkout ref for the reviewer, posted inline.
There was a problem hiding this comment.
Verdict: 1 Low
Onboarding-only PR: four bot workflows + two local composite actions + .bot/ config/prompts, no connector source touched. Auth (engine-scoped App token via git extraheader, PAT-free), input hardening (digits-only validation, untrusted issue body → file, heredoc-delimiter collision check), and the fork/open-PR/label gates all look sound. One low finding: the actually-pinned engine SHA (d24ca217...) matches nowhere in the PR description or design doc (which cite b6205fb / d780b2d), and is duplicated across all four workflows.
There was a problem hiding this comment.
Verdict: 1 Low
Looks good — bot/CI onboarding only, no connector source touched; auth flow, fork gates, and untrusted-input handling are careful and well-commented. One low inline note: engineer-bot-followup.yml passes PR_BASE_SHA but (unlike reviewer-bot-followup.yml) never ensures the base commit is fetched. Meta nit for the author, not inline: the PR description states the engine is pinned to SHA b6205fb, but all four workflows pin d24ca2171d191a652a67f4f43995f0959c9a5791 (the design doc's later update confirms b6205fb was an earlier draft) — update the description so reviewers/maintainers see the engine version that actually runs.
There was a problem hiding this comment.
Verdict: 1 Low
Solid, carefully-written bot-onboarding infra — vendored code is fully removed (no dangling refs outside the design doc), setup-poetry supplies the interpreter the engine install needs, and security handling (40-char SHA pin validation, masked tokens, git extraheader instead of in-URL creds, argv-prefix bash allowlist, untrusted-input-to-file) is thorough. One low error-handling nit inline. Nits (summary-only): the PR title/description states the engine is pinned to b6205fb, but all four workflows pin d24ca2171d191a652a67f4f43995f0959c9a5791 (the description's later "Update" sections clarify b6205fb was an earlier draft — worth aligning the title). Similarly, the design doc's "Engine pin" section still names d780b2da… while its own later updates use d24ca217…; harmless but stale.
There was a problem hiding this comment.
Verdict: 1 Medium
Solid, carefully-gated onboarding PR — no connector source changes, and the workflow/token/security logic is well-reasoned and thoroughly commented. One medium consistency gap: the .bot/config.yaml templates (name, branch_prefix, pr_title_template, commit_message_template) are all bug-fix-only, but engineer-bot.yml now routes Feature issues to the enhancement flow, which would emit fix:/bugfix-labelled PRs, commits, and branches for enhancements.
There was a problem hiding this comment.
Verdict: 1 Low
Infrastructure onboarding PR (bot workflows, two local composite actions, engineer-bot config + prompts, design doc). Well-structured and heavily annotated with security reasoning; the token minting, digits-only issue/PR validation, $GITHUB_ENV heredoc-delimiter guard, SHA-pin validation, and fork/open-PR gates all look sound, and no dangling references to the removed vendored bot code remain in the workflows. One low note on the git diff bash-allowlist claim; otherwise looks good.
There was a problem hiding this comment.
Verdict: 1 Low
Solid, security-conscious bot-onboarding infra PR — no connector source changes, so no unit-test gap. One low finding: the committed design doc / PR description are stale vs the shipped code on the pinned engine SHA (d24ca21 documented vs d9cb1e6 actually pinned in all four workflows) and the Feature→flow mapping (enhancement documented vs task in code). Two acknowledged items I did not file as findings: the workflow_dispatch reviewer path checks out the default branch rather than the PR head (already called out as a tracked follow-up in the PR description), and the engine-ref SHA is hand-duplicated across four workflows with only a comment enforcing lockstep bumps (a nit — a shared var/CI check would remove the drift risk).
There was a problem hiding this comment.
Verdict: 1 Low
Looks good — workflow/config/docs onboarding only, no src/ changes, and the four bot workflows are internally consistent (same engine SHA across all four, satisfying the documented lockstep requirement). One low: the design doc and PR description cite engine SHAs (d780b2d, d24ca21) that don't match the d9cb1e63… actually pinned in the workflows. Security-sensitive paths (token minting/masking, fork gates, digits-only input validation, heredoc-delimiter collision guard, argv-prefix bash allowlist) are well-reasoned and correctly implemented.
There was a problem hiding this comment.
Verdict: 1 Medium
Bot-onboarding infra PR (workflows + .bot/ prompts/config); no product code changes. The security gating, token minting, and shell-injection guards are carefully handled. One medium concern: the author flow can be task (for Feature/Task issues) while the engineer prompts are written only for the bug-fix flow, giving the agent contradictory instructions.
There was a problem hiding this comment.
Verdict: 1 Low
Workflow/config-only onboarding PR (no src/ changes); gates, token-minting, and JFrog/OIDC install are carefully built and the security rationale is well-documented. One low note: the engine SHA is duplicated across all four workflows and could be centralized in bot-prelude's engine-ref default. Nit: the PR description says the engine is pinned to d24ca21, but the workflows (and the design-doc update) actually pin d9cb1e63… — worth reconciling the description so the stated pin matches what ships.
There was a problem hiding this comment.
Verdict: 1 Low
Looks good — infra-only PR (workflows, composites, .bot/ config + prompts) that is carefully written and, per the design doc, already exercised live for both bots. Verified setup-poetry exists with the expected python-version input and that no dangling references to the removed vendored bot code remain under .github. Only one low-severity note: the engine SHA actually pinned in bot-prelude differs from the SHAs quoted in the PR description/design doc — reconcile the docs.
There was a problem hiding this comment.
Verdict: 1 Low
Looks good — a clean, carefully-constructed CI onboarding PR (workflows + local composites + .bot/ prompts/config; no executable connector code changed). Gates, untrusted-input handling, and token scoping are thorough; I verified the vendored bot code is fully removed with no dangling references and that setup-poetry / the referenced composites line up. One low-severity credential-cleanup asymmetry noted inline. Note: the PR description still cites engine pin d24ca21, but the actual bot-prelude default (and the design-doc banner) is 3963f4be76be3b4ac0382ca0d7b6bf9d051929e6 — the code is internally consistent, but the description is stale.
There was a problem hiding this comment.
Verdict: 1 Medium
Looks solid overall — this is well-reasoned bot-onboarding infra (workflows + composites + prompts), with no src/ changes so no unit-test coverage is expected. One medium defense-in-depth gap: the git diff HEAD / git log bash-allowlist prefixes still permit an arbitrary-path write via --output=<abs>, which contradicts the file's own carefully-argued sandbox model (see inline finding).
External consumers can't `uses:` the internal engine's actions cross-repo, so install the engine via a LOCAL composite that pip-installs it over HTTPS with a short-lived, engine-scoped GitHub App token (no stored PAT), routed through the internal JFrog mirror. bot-prelude wraps the shared per-workflow setup (mint bot token + engine-scoped token + Node + install) and centralizes the engine pin in one place — engine-ref default, pinned to databricks-bot-engine@3569dae (post Type-flow #107, REPO_ROOT #115, DCO sign-off #117). Signed-off-by: Eric Wang <e.wang@databricks.com> Co-authored-by: Isaac Signed-off-by: eric-wang-1990 <e.wang@databricks.com>
Reviewer + reviewer-followup as own-job workflows that install the engine via the local bot-prelude composite (PAT-free App-token auth) and run reviewer_bot.run_review / .followup. Triggers on every non-fork PR; fork gate in each job `if:`. .bot/prompts/review/system.md is repo-specific ADDITIVE guidance appended to the engine's base reviewer prompt. Signed-off-by: Eric Wang <e.wang@databricks.com> Co-authored-by: Isaac Signed-off-by: eric-wang-1990 <e.wang@databricks.com>
Engineer author (label an issue) + followup (review-comment replies) as own-job workflows: setup-poetry for the connector's test deps, engine install via the local bot-prelude composite, then engineer_bot.run. Author flow selected from the issue Type (Bug ⇒ bug-fix; other/none ⇒ .bot/config.yaml default, also bug-fix). REPO_ROOT points the agent's working tree + .bot/ lookup at the checkout. .bot/config.yaml (bug-fix, poetry pytest allowlist, flow-aware templates omitted to use engine defaults) + engineer/engineer-followup prompts. Verified live — reviewer posts findings; engineer author + followup run end to end. docs/superpowers/specs/: onboarding design + first-consumer findings trail. Signed-off-by: Eric Wang <e.wang@databricks.com> Co-authored-by: Isaac Signed-off-by: eric-wang-1990 <e.wang@databricks.com>
Summary
Onboards both bots on
databricks-sql-pythonto the upstreamdatabricks/databricks-bot-engine, pinned to engine SHAd24ca21(post-#107 —author flow selected from the issue Type) and fully PAT-free (App-auth).
This repo is the first clean consumer of the engine; onboarding friction is
captured as feedback (see the design doc + engine #97).
Design doc:
docs/superpowers/specs/2026-07-14-onboard-bot-engine-design.md.Reviewer bot (was vendored → now engine reusables, App-auth)
scripts/reviewer_bot+scripts/sharedwith thincallers of the engine's
reviewer-bot.reusable.yml/-followup.reusable.yml.engine-auth: app+ noengine-pat: the reusable mints an engine-scoped Apptoken from the review-bot App creds (engine PR sqlalchemy > 2.0.0 #100).
Engineer bot (new — flow from issue Type)
engineer-bot.yml(author, label an issue) +engineer-bot-followup.yml(reply on an
engineer-bot-labeled PR). Own jobs (they build/run theconnector), each minting an engine-scoped App token for a PAT-free REF install.
Bug⇒bug-fix,Feature⇒enhancement, any other type / none ⇒ the.bot/config.yamlflow: bug-fixdefault.environment: azure-prod; explicitpoetry installso the agent can runpoetry run python -m pytest tests/unitto self-verify its red→green fix..bot/config.yaml+ engineer prompts, ported from the engine dogfood andrewritten for the connector (
src/databricks/sql/, mockedtests/unit).Secrets (all provisioned)
REVIEW_BOT_APP_ID/PRIVATE_KEY,ENGINEER_BOT_APP_ID/PRIVATE_KEY,DATABRICKS_HOST,DATABRICKS_TOKEN. NoBOT_ENGINE_PAT— App-authreplaces it.
Prerequisite (engine-repo admin) — ✅ done
The review-bot and engineer-bot GitHub Apps are installed on
databricks-bot-enginewithcontents: read(required, else the App-authengine install fails for both bots).
Verification
.bot/config.yamlparse;.bot/tree complete; vendoredcode fully removed with no dangling references.
engine-auth: appinstall path validated end-to-end (engine PR sqlalchemy > 2.0.0 #100).workflow_dispatch(manual) reviewer path theexplore-tool read root falls back to the default branch — the automatic
pull_requestpath (the normal case) is unaffected. Tracked separately.This pull request and its description were written by Isaac.